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Problem  and  Introduction 

Continues  work  reported  at  2009  TSP  Symposium 

309th  SMXG  (Hill  AFB)  has  had  a  long-term  initiative  to  improve 
performance  based  on  CMMI®  and  TSP®. 

Work  Progress 

•  Multi-year  effort  with  SEI  support  and  internal  resource  commitments 

•  Training  in  the  basic  SEI  technologies  plus  extensive  training  in  six-sigma 
methods. 

Plan:  sustain  and  improve 

Goal:  Improve  quality,  cost  and  schedule  performance. 

•  What  can  we  monitor  in-process?  How? 

•  Can  we  demonstrate  that  team  performance  factors  contribute  to  the  desired 
program  outcomes? 

•  Which  practices  are  the  most  influential? 
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Problem  Model 

Robust  analysis  of  cause  and  effect 


Survey  Q14 
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Legend: 

Box=  Observable  &  Measurable  Factor 
Oval=Latent  factors  (not  directly  measurable) 
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Solution  Approach 


Use  team-based  surveys  to  assess  activities,  attitudes  and  beliefs. 

•  Design  surveys  to  reference  essential  TSP  concepts. 

•  Data  collection  frequency  aligned  to  frequency  of  TSP  events: 

-  launch,  weekly  meetings,  monthly,  quarterly  and  annual  reviews 

•  Minimize  cost  to  team  by  sampling  strategy,  limiting  questions,  limiting  list  of 
possible  answers. 

•  Minimize  cost  to  team  by  using  Quatrics,  web-based  surveys. 

Prepare  for  data  analysis. 

Report  analysis  to  squadron  leaders  and  group  staff  on  quarterly  basis. 

Total  number  of  survey  questions  =  105! 
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Survey  Design  and  Implementation 

Surveys  began  in  January  2014 
Sampling 

•  Randomly  select  sample  from  each  team  so  no  individual  feels  burdened. 

•  No  one  should  receive  more  requests  than  8/year 

Survey  Design 

•  Five  online  surveys  based  on  type  of  activity  monitored  (e.g.  launch). 

•  Tested  and  verified  questions  with  staff,  (yes/no  answers) 

Implementation 

•  Obtained  Qualtrics  license  to  create  secure,  anonymous  survey  links. 

•  Code  survey 

•  Initiate  surveys  by  team  schedule  (began  with  weekly  and  monthly  ones) 

•  Response  data  analyzed  with  Excel 

Analysis 

•  Performed  jointly  by  SEI  and  309th  SMXG 

Report  at  both  squadron  and  group  level 
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Nature  of  Team  Survey  Concerns 

Observations  and  Assessment  of  Performance  and  Behavior 


Leadership 

•  Clarity  of  goals,  quality  suffering,  employee  role  satisfaction,  motivation,  team 
member’s  satisfaction,  prioritization  of  work,  measurement  of  performance, 
team  decisions,  change  in  direction,  team  conflicts,  poor  performance  from 
members 

Operational 

•  Load  balancing,  not  following  process,  meetings,  progress  reviews,  no  impact 
analysis,  consensus  on  team  decisions,  ideas  for  improvement,  improvement 
data  collection,  stakeholder  involvement,  number  of  tasks,  ask  for  help, 
customer  involvement,  checklists,  problem  tracking,  collection  of  data, 
communication,  stress  and  overtime 

Cultural 

•  Face  time,  team  cooperation,  individual  commitment,  submit  ideas  for 
improvement 
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Nature  of  Survey  Questions 
(the  outcomes) 


Questions  about  Perceived  Performance  (were  asked  at  the  end  of  each 
survey) 

Qi 3.  Describe  your  team  performance  at  this  point  in  lime 

Worse  man  Plan  On  Plan  Bolter  man  Plan  Don't  Know 

Quality  Q  ©  0 

Cost 

Schedule 


Intent  is  to  analyze  the  ability  to  predict  these 
outcome  measures  based  on  the  preceding 

leading  indicators 
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Response  Rates 


The  Weekly  Survey  Response  Rate  has  been  about  50% 
The  Monthly  Survey  Response  Rate  has  been  about  43% 
The  Quarterly  Survey  Response  Rate  has  been  about  40% 
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Some  Interesting  Overall  Question  Results 


Yes 

No 

Did  your  team  leader  properly  deal  with  any  poor  performers  on  your  team?  (5) 

68% 

31% 

Were  you  aware  of  team  conflicts  that  were  not  resolved  in  a  timely  manner?  (6) 

14% 

85% 

Were  you  aware  of  team  members  unhappy  with  their  assigned  work?  (11) 

33% 

65% 

Were  you  aware  of  important  team  decisions  that  were  missed  or  late?  (12) 

29% 

69% 

Did  your  team  reach  consensus  when  needed  on  key  team  decisions?  (5) 

85% 

14% 

Did  you  collect  your  own  personal  quality  data?  (28) 

44% 

53% 

Did  you  observe  team  members  under  unusual  stress  or  working  excessive  overtime?  (31) 

28% 

69% 

Was  there  an  open  climate  to  submit  ideas  for  improvement?  (1) 

87% 

13% 

Was  open  discussion  and  individual  commitment  demonstrated  within  your  team?  (2) 

89% 

11% 

Were  you  satisfied  with  the  degree  of  internal  team  cooperation?  (3) 

87% 

12% 

Were  you  satisfied  with  the  degree  of  one-to-one  face  time  you  experienced  with  your 
team  leader?  (4) 

85% 

14% 

Almost  all  questions  depicted  statistically  significant  differences 

among  squadrons 
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Results  -  01 

Questions  that  were  most  significantly  related  to  the  quality  outcome: 

•  Were  you  satisfied  with  the  quality  and  timeliness  of  team  improvement  data 
collected? 

•  Were  you  satisfied  with  the  frequency  and  nature  of  team  communications? 

•  Were  you  satisfied  with  the  degree  to  which  process  problems  were  tracked, 
handled  and  resolved  in  a  timely  fashion? 

•  Was  there  an  open  climate  to  submit  ideas  for  improvement? 

•  Were  you  aware  of  quality  suffering  to  meet  other  goals?  (No) 

•  Were  you  satisfied  with  the  degree  to  which  process  guidance  and  checklists 
were  used? 

•  Were  you  satisfied  with  the  degree  of  internal  team  cooperation? 

•  Were  you  unclear  about  any  of  the  primary  team  goals?  (No) 

•  Were  you  dissatisfied  with  the  number  of  recurring  issues  faced  by  the  team? 

•  Were  you  satisfied  with  the  actual  versus  planned  number  of  quality  reviews 
held  within  your  team? 

•  Were  you  satisfied  with  the  action  taken  based  on  quality  measures  collected 
within  your  team? 
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Results  -  02 


Question  that  was  most  significant  with  the  schedule  outcome: 

•  Were  you  satisfied  with  how  your  team  prioritized  work? 

•  Were  you  dissatisfied  with  the  number  of  recurring  issues  faced  by  the  team? 

•  Were  you  satisfied  with  the  action  taken  based  on  quality  measures  collected 
within  your  team? 

Questions  that  were  significant  with  the  cost  outcome: 

•  Were  you  dissatisfied  with  the  number  of  recurring  issues  faced  by  the  team? 

•  Were  you  dissatisfied  with  the  team's  working  conditions  or  available  tools 
and  technology?  (No) 
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Proportion  for  Q9_5Yes 


Statistical  Process  Control  Chart  (p  chart)  for 
Each  Survey  Question 

Enables  each  question  to  become  an  early  warning  radar! 


i.i 
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Future  Work 


Improve  decision-making  and  intervention 

•  Trim  uninteresting  or  non-value  added  questions  from  survey. 

•  Transition  from  binary  (yes/no)  to  scaled  responses  for  more  robust  analysis. 

•  Implement  more  robust  statistical  (structural  equation)  model  for  outcomes  of 
perceived  cost,  schedule  and  quality 

•  Collect  actual  outcome  measure  data  and  repeat  statistical  analysis. 

Develop  materials  for  broader  use  and  more  specific  questions 

•  Encourage  other  teams  and  organizations  to  use  these  survey  questions  and 
share  data  for  benefit  of  community  benchmarks  and  heuristics  for  superior 
team  performance 
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Structural  Equation  Model 

Robust  analysis  of  cause  and  effect 


Observable 
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Legend: 

Box=  Observable  &  Measurable  Factor 
Oval=Latent  factors  (not  directly  measurable) 
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Summary 


Survey  approach  appears  to  be  feasible  from  both  response  rates,  data 
collection  and  value-added  results  of  significant  factors 


More  time  needed  to  demonstrate  ability  of  the  control  charts  to  provide 
valuable  early  warning  on  changes  in  team  operations 


Other  services  are  showing  interest  in  these  survey  questions  and  have 
begun  the  journey  of  implementing  the  survey  questions 


Expect  cross  service  benchmarks  to  share  next  year! 


Overtime,  results  could  serve  to  improve  TSP  Coach  training 
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Contact  Information 


Robert  W.  Stoddard 
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Software  Solutions  Division,  SEI 
Telephone:  +1  412-268-1121 
Email:  rws@sei.cmu.edu 

Web 

www.sei.cmu.edu 

www.sei.cmu.edu/contact.cfm 


U.S.  Mail 

Software  Engineering  Institute 
Customer  Relations 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-2612 
USA 

Customer  Relations 

Email:  info@sei.cmu.edu 
Telephone:  +1  412-268-5800 
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